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ABSTRACT 

Graduates of the service academies operated by the Army, 

Navy, and Air Force currently make up approximately 18 percent of the officer 
corps for the nation's armed services. The academies represent the military's 
most expensive source of new officers. The Department of Defense (DOD) pays 
the full cost of a student's 4-year education at the academies/ and the 
related cost has increased over the past 4 years. Admission to the academies 
is highly competitive. The academies use a "whole person" method to make 
admission decisions. Recent studies by the Air Force raised questions about 
possible adverse effects of whole person admissions policies on student 
quality. GAO was asked to review all three service academies and specifically 
address the extent to which: (1) DOD oversees the service academies; (2) 

applicants are granted waivers of academic standards; and (3) various groups 
of students differ in admissions scores and academy per f ormance . The Office of 
the Under Secretary of Defense for Personnel and Readiness (OUSD/P&R) , the 
services, and the academies' boards of visitors conduct considerable 
oversight of the academies' operations and performance, but they lack a 
complete oversight framework. A complete oversight framework includes 
performance goals and measures against which the academies' performance could 
be better assessed. OUSD/P&R and the services use the number and type of 
commissioned officers as the primary measure of academy performance. OUSD/P&R 
requires and receives reports on academy performance from the services. While 
data submitted in these reports provide perspective on current performance 
compared with past performance, without stated performance goals and 
measures, these reports do not offer OUSD/P&R or the services as good an 
insight into the academies' performance as they could. Additionally, though 
the academy boards of visitors serve as an external oversight mechanism to 
focus attention on a wide range of issues, they also do not assess the 
academies' performance against established performance goals and measures. 

The academies do not grant waivers from academic criteria or have absolute 
minimum scores for admission. However, under the whole person approach, the 
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academies can admit some applicants whose academic scores are lower than 
might normally be competitive for admission, but who in their totality 
(academics, physical aptitude, and leadership) are evaluated by academy 
officials as being capable of succeeding at the academy. In reviewing the 
academy classes that started in 1998 (class of 2002), it was found that 
despite differences among various groups of students in their admissions 
scores and similar differences in their performance while at the academies, 
the differences in performance were not sizable. Some groups, such as 
females, performed better in some categories than the class as a whole and 
worse in others. Some groups (minorities, preparatory school graduates, 
recruited athletes, and students in the lower 30 percent of their class in 
terms of academic admissions scores) performed at lower levels on average in 
all categories than the class as a whole. (Author/EV) 
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The Office of the Under Secretary of Defense for Personnel and Readiness 
(OUSD/P&R), the services, and the academies’ boards of visitors conduct 
considerable oversight of the academies’ operations and performance, but 
they lack a complete oversight framework. A complete oversight framework 
includes performance goals and measures against which the academies’ 
performance could be better assessed. OUSD/P&R and the services use the 
number and type of commissioned officers as the primary measure of 
academy performance. OUSD/P&R requires and receives reports on 
academy performance from the services. While data submitted in these 
reports provide perspective on current performance compared with past 
performance, without stated performance goals and measures, these reports 
do not offer OUSD/P&R or the services as good an insight into the academies 
performance as they could. Additionally, though the academy boards of 
visitors serve as an external oversight mechanism to focus attention on a 
wide range of issues, they also do not assess the academies’ performance 
against established performance goals and measures. 

The academies do not grant waivers from academic criteria or have absolute 
minimum scores for admission. However, under the whole person approach, 
the academies can admit some applicants whose academic scores are lower 
than might normally be competitive for admission, but who in their totality 
(academics, physical aptitude, and leadership) are evaluated by academy 
officials as being capable of succeeding at the academy. 

In our review of the academy classes that started in 1998 (class of 2002), we 
found that despite differences among various groups of students in their 
admissions scores and similar differences in their performance while at the 
academies, the differences in performance were not sizable. Some groups, 
such as females, performed better in some categories than the class as a 
whole and worse in others. Some groups (minorities, preparatory school 
graduates, recruited athletes, and students in the lower 30 percent of their 
class in terms of academic admissions scores) performed at lower levels on 
average in all categories than the class as a whole. 



Academy Operating Costs and Cost Per Graduate, Fiscal Years 1999-2002 


Academy 


Cost category 


FY1999 


FY 2000 


FY 2001 


FY 2002 


Military 


Total operating 
costs 


$301,058,452 


$330,603,820 


$336,416,716 


$364,971,975 


Academy 


Cost per graduate 


312,150 


320,120 


339,318 


349,327 


Naval 


Total operating 
costs 


245,749,679 


253,817,467 


273,809,865 


292,696,358 


Academy 


Cost per graduate 


254,983 


256,931 


266,033 


275,001 


Air Force 


Total operating 
costs 


277,639,005 


314,972,559 


321,335,152 


333,056,023 


Academy 


Cost per graduate 


305,945 


305,133 


313,456 


322,750 



Source: DOD. 
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Accountability * Integrity * Reliability 



United States General Accounting Office 
Washington, DC 20548 



September 10, 2003 

The Honorable Jerry Lewis 
Chairman 

The Honorable John P. Murtha 
Ranking Minority Member 
Subcommittee on Defense 
Committee on Appropriations 
House of Representatives 

Graduates of the service academies operated by the Army, Navy, and Air 
Force make up approximately 18 percent of the officer corps for the 
nation’s armed services.’ The academies represent the most expensive 
source of new officers, compared with other sources for officers, such as 
Reserve Officer Training Corps (ROTC) progreuns at colleges and 
universities or officer candidate/training schools for individuals who 
already have college degrees. The Department of Defense (DOD) pays the 
full cost of providing the 4-year progreuns of academic education, military 
training, physical conditioning, and pay for each student.^ In fiscal year 
2002, DOD reported costs per graduate for the U.S. Military Academy, the 
U.S. Naval Academy, and the U.S. Air Force Academy were approximately 
$349,000, $275,000, and $333,000, respectively. These costs have increased 
over the past 4 years. To ensure the best value for the investment in the 
academies, effective management principles are critical. Such principles 
include a complete oversight freunework, with clear roles and 
responsibilities, as well as performance goals and measures against which 
to objectively assess performance. 

With each academy accepting about 1,200 of its more than 10,000 
applicants a year, admission to the academies is highly competitive. 
Applicants must be selected or obtain a nomination, such as from a 
senator, representative, the President, or the Vice President, based on the 



' The Marine Corps does not have its own academy. The Naval Academy graduates both 
Navy and Marine Corps officers. 

^ Students attending the U.S. Military Academy at West Point, New York, and the U.S. Air 
Force Academy in Colorado Springs, Colorado, are called “cadets,” while those attending 
the U.S. Naval Academy, in Aimapolis, Maryland, are called “midshipmen.” We refer to 
cadets and midshipmen collectively as “students.” 
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categories established by law.’ Most nominations are reserved for 
Congress, which, therefore, has a central role in admitting students to the 
academies. In addition to basic age and medical qualifications, the 
academies’ admissions process involves an assessment of applicants’ 
academic achievement (e.g.. Scholastic Aptitude Test — SAT — scores and 
grade point averages), physical aptitude, and extracurricular activities 
(i.e., leadership potential). Academy officials combine these assessments 
into a “whole person” admissions score that is used to determine an 
applicant’s potential to graduate from an academy and potential fitness as 
a commissioned officer. Apphcants compete for admission based on these 
scores. 

Air Force studies have raised questions about possible adverse effects of 
whole person admissions pohcies on student quahty. For example, the Air 
Force found that its whole person assessments and resulting admissions 
scores have led the Air Force Academy to admit an increasing number of 
students whose academic quahfications are below academic minimums, as 
well as to admit an increasing number of students recruited largely to 
participate in varsity intercollegiate athletics. 

The House of Representatives report on defense appropriations for fiscal 
year 2003^ directed that we perform reviews of all three service academies 
and their respective preparatory schools.’ As part of the review of the 
service academies, we were also directed to obtain student and faculty 
perceptions of various aspects of student life at the academies. Issues 
associated with the academy preparatory schools and the results of 
surveys on aspects of student life are addressed in separate reports.’ 



’ 10 U.S.C. §§ 4342, 6954, and 9342. 

^ H.R. Rept. 107-532, at 14-15 (2002). 

’ The academy preparatory schools exist to prepare selected students who are not ready 
academically to attend one of the academies. 

® U.S. General Accounting Office, Military Education: DOD Needs to Align Academy 
Preparatory Schools’ Mission Statements with Overall Guidance and Establish 
Performance Goals, GAO-03-1017 (Washington, D.C.: September 2003); and Military 
Education: Student and Faculty Perceptions of Student Life at the Military Academies, 
GAO-03-1001 (Washington, D.C.: September 2003). 
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As agreed with yovir offices, this report addresses the following questions, 
to what extent 

(1) does DOD oversee the academies’ operations and performance? 

(2) are apphcants granted waivers from academic criteria for admissions? 

(3) do various groups of students differ in admissions scores and academy 
performance? 

In addition to reviewing documents and interviewing officials at all three 
academies, the service headquarters, the Office of the Under Secretary of 
Defense for Personnel and Readiness (OUSD/P&R), and the academies’ 
boards of visitors, we reviewed admissions policies and procedures and 
observed their use by academy officials in evaluating apphcations being 
considered for the incoming class of 2007. We also obtained and analyzed 
admissions and performance data for the student class that graduated in 
2002. To compare student admissions qualifications and performance at 
the academies, we identified six mgjor groups of students common to all 
academies: females, minorities, academy preparatory school graduates, 
recruited athletes, prior enlisted personnel, and students whose academic 
admission scores fell in the lower 30 percent of the entering class.^ Data on 
student performance included academic grade point average; military 
performance average, which is similar to a performance evaluation for 
commissioned officers; and class rank.® It also included graduation rate. 
Other issues, such as recent controversies associated with alleged sexual 
assault, did not fall within the scope of this review. Further details on our 
scope and methodology are in appendix I. We conducted our work from 
October 2002 through May 2003 in accordance with generally accepted 
government auditing standards. 



Results in Brief 



The Office of the Under Secretary of Defense for Personnel and Readiness 
(OUSD/P&R), the services, and the academies’ boards of visitors conduct 
considerable oversight of the academies’ operations and performance, but 
they lack a complete oversight framework. In 1991, our report concluded 
that better oversight of the academies was needed and made 
recommendations to improve DOD oversight. Since then, DOD has taken 



^ Each group may contain members of the other groups. 

® Class rank is referred to as “order of merit” by the academies. 
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measures to address these issues, including establishing guidance on 
oversight of the academies and uniform academy cost reporting. However, 
DOD has not established a complete oversight framework, which would 
include not only clear roles and responsibilities, but also performance 
goals and measures against which to objectively assess performance. 
OUSD/P&R, the services, and the academies^ boards of visitors have 
different oversight roles, but largely conduct oversight activities without 
the benefit of formalized performance goals and measures. OUSD/P&R 
and the services use the number and types of commissioned officers as the 
primary measure of academy performance. OUSD/P&R requires and 
receives reports on academy performance from the services. 

While data submitted in these reports provide perspective on such 
performance measures as graduation rates, admissions trends for women 
and minorities, and information on the quality of admitted students, 
without stated performance goals and measures, these data do not offer 
OUSD/P&R or the services as good an insight into the academies' 
performance as they could. For example, the data collected by the 
academies show that the graduation rates have increased in the last 10 
years; however, there is no stated goal for graduation rate against which to 
judge whether this rate of increase is adequate. Other data collected by the 
academies indicate that the percentage of females and minorities has 
fluctuated over the last 3 years, but apart from admissions targets used by 
the Military Academy, there are no stated goals against which to measure 
the adequacy of these admissions trends. Additionally, academy officials 
regularly analyze data on student performance to determine the extent to 
which admissions standards can be changed to improve overall student 
performance at the academies. However, there are no stated goals for 
student body performance, apart from minimiun graduation standards 
such as the cumulative academic grade point average, that might help the 
academies and other oversight bodies assess overall student performance. 
Additionally, each academy’s board of visitors — an external oversight 
mechanism — focuses attention and actions on a wide range of operational 
and quality of life issues at the academies. However, the boards do not 
evaluate academy performance against established performance goals and 
measures. Without formal goals and measures that are, moreover, linked 
to mission statements, oversight bodies do not have sufficient focus for 
their efforts and cannot systematically assess an organization’s strengths 
and weaknesses nor identify appropriate remedies that would help them 
achieve the best value for the nation’s investment in the academies. 

The academies do not grant waivers from academic criteria or have 
absolute minimum scores for admission. Under the whole person 
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approach, the academies can admit some applicants whose academic 
scores are lower than might normally be competitive for admission, but 
who in their totality (academics, physical aptitude, and leadership) are 
evaluated by academy officials as being qualified and capable of 
succeeding at the academy. The only admissions criteria with an absolute 
minimum score for qualifying for admissions is physical aptitude. The 
academic and leadership criteria have a range of qualifying scores based 
on what general levels of ability are considered competitive during the 
admissions process. If an applicant’s score is lower than the competitive 
range in academics, then admission officials have some flexibility in 
(1) further considering the applicant by re-examining the student’s record 
for information that can produce further insight about his or her academic 
achievement and (2) weighing the extent to which the leadership 
component of the whole person score may offset the low component. It is 
possible for students to be admitted whose academic scores were not as 
competitive as some of their peers who may not have been admitted. The 
applicant is considered a risk and is evaluated through a deliberative 
process by academy officials on the basis of their judgment of whether the 
applicant is fully qualified and capable of succeeding at that academy. The 
subjective nature of this approach is consistent with the intent of the 
whole person concept, by which the academies want to admit students 
who also demonstrate leadership characteristics that cannot be quantified 
by purely objective scoring methods. Academy officials do not consider 
this approach to represent an academic waiver, but instead their judicious 
assessment of the whole person. 

In our review of the academy classes that started in 1998 (class of 2002), 
we found differences among various groups of students in their 
admissions scores and similar differences in their performance while at 
the academies; the differences in performance were not sizable. For the 
class data we reviewed, minorities, academy preparatory school 
graduates, recruited athletes, and prior enlisted students® all had lower 
average admissions scores than the average for the class as a whole. Of 
those students in the lower 30 percent of the class in terms of academic 
admissions scores, about 44 percent were recruited athletes, between 25 
and 31 percent were minorities, and between 20 and 34 percent were 
preparatory school graduates. Regarding performance, we found 
differences at the academies between selected groups (i.e., females, 
minorities, preparatory school graduates, recruited athletes, prior enlisted 



® Each of these groups can contain members from other groups. 
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students, and students in the lower 30 percent of the class in terms of 
academic admissions scores) and the class as a whole. Those differences 
varied but were generally not sizable. For example, females at one 
academy had a lower graduation rate than the class as a whole but a 
higher average academic grade point average and a higher average class 
rank. Some groups at all academies — such as minorities, preparatory 
school graduates, recruited athletes, and students in the lower 30 percent 
of their class in terms of academic admissions scores — performed at lower 
levels on average in all categories than the class as a whole, but these 
differences were not significant. For example, one of the lowest average 
academic grade point averages among the groups we reviewed was 2.61, 
whereas the average for the class as a whole at that academy was 2.93, A 
2.0 grade point average is required to graduate. The lowest graduation rate 
for the class we reviewed was 65 percent for the students in the lower 30 
percent of their class in terms of academic admissions scores at one 
academy. The average graduation rate for the class as a whole at that 
academy was 74 percent. 

We are making a recommendation to improve DOD’s oversight of 
operations and performance at the academies through the enhancement of 
performance goals and measures. In comments on a draft of this report, 
DOD concurred with our recommendation. 



Background 



The Departments of the Army, the Navy, and the Air Force each have their 
own educational institutions (academies) to produce a portion of each 
branch’s officer corps: 

• U.S. Military Academy (West Point, N.Y.), established in 1802; 

• U.S. Naval Academy (Annapolis, Md.), established in 1845; and 

• U.S. Air Force Academy (Colorado Springs, Colo.), established in 1954. 

The academies are structured to provide a curriculum critical to the 
development of successful future officers in academic, military, and 
physical areas of achievement. Additionally, the academies emphasize the 
moral and ethical development of students through their respective honor 
codes and concepts. 



Other sources for commissioned officers include ROTC programs at colleges and 
universities and officer candidate/training schools for individuals who already have college 
degrees. 
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There are approximately 4,000 students eruoUed at each of the three 
service academies at any given time, each comprising four classes. In 
December 2002, Congress authorized an annual increase of up to 100 
students until the total munber reaches 4,400 for each academy. “ In 2002 
the Mihtary Academy graduated 968 students; the Naval Academy 977 
students; and the Air Force Academy 894 students. Faculty at the U.S. 
Mihtary Academy and the U.S. Air Force Academy are comprised 
predominantly of mihtary officers (79 and 75 percent, respectively), whhe 
at the U.S. Naval Academy 59 percent of the faculty are civhians. Table 1 
shows the composition of the faculty at the service academies. 



Table 1 : Civilian and Military Faculty at the Service Academies 


Service academy 


Total 
number 
of faculty 


Total number of 
civilian faculty 
(% of faculty) 


Total number of 
military facuity 
(% of faculty) 


U.S. Military Academy 


622 


131 (21%) 


491 (79%) 


U.S. Naval Academy 


555 


326 (59%) 


229 (41%) 


U.S. Air Force 
Academy 


490 


123 (25%) 


367 (75%) 


Total 


1,667 


580 (35%) 


1,087 (65%) 



Source: DOD. 

Note: Faculty information is based on a snapshot of each academy in February 2003. 



DOD reports that the total cost to operate all three academies in fiscal 
year 2002 was $990.7 million. Table 2 shows the reported operating costs 
and cost per graduate for each academy from fiscal year 1999 through 
fiscal year 2002. We did not independently verify these costs. 



" Pub. L. 107-314, Bob Stump National Defense Authorization Act for Fiscal Year 2003^ § 
532, December 2, 2002. 




Page 7 



13 



GAO-03-1000 Military Education 



Table 2: Academy Operating Costs and Cost Per Graduate, Fiscal Years 1999-2002 



Academy 


Cost Category 


Fiscal year 
1999 


Fiscal year 
2000 


Fiscal year 
2001 


Fiscal year 
2002 


Military Academy 


Total operating costs 


$301,058,452 


$330,603,820 


$336,416,716 


$364,971,975 




Cost per graduate 


312,150 


320,120 


339,318 


349,327 


Naval Academy 


Total operating costs 


245,749,679 


253,817,467 


273,809,865 


292,696,358 




Cost per graduate 


254,983 


256,931 


266,033 


275,001 


Air Force Academy 


Total operating costs 


277,639,005 


314,972,559 


321,335,152 


333,056,023 




Cost per graduate 


305,945 


305,133 


313,456 


322,750 



Source: DOD. 



Prospective students must meet basic eligibility requirements for 
appointment to an academy. They must (1) be unmarried, (2) be a U.S. 
citizen, (3) be at least 17 ye^u•s of age and must not have passed their 
twenty-third birthday on July 1 of the ye^u• they enter an academy, (4) have 
no dependents, and (5) be of good moral ch^u•acter.^^ 

After determining eligibility, a candidate submits an application to a 
preferred academy or academies. Each submitted application is required 
to include information such as, but not limited to, the candidate’s (1) SAT 
scores (or American College Testing — ACT — examination scores); (2) high 
school grade point average (and class rank, if possible); (3) physical 
aptitude scores; (4) medical examination results; and (5) extracurricular 
activities. The academies admit those candidates that have secm-ed a 
nomination and who represent, in the opinion of academy officials, the 
best mixture of attributes (academic, physical, and leadership) necessary 
to ensure success at the academies and as military officers. 

The military academies use a “whole person” method to assess potential 
candidates in three major areas: (1) academics, (2) physical aptitude, and 
(3) leadership potential. Each academy uses the same basic approach. 
Admissions assessments are weighted toward academic scores that 
include objective tests and high school performance. Leadership potential 
is measured by assessing athletic and non-athletic extracurricular 
activities. Subjective assessments of potential candidates in these major 
areas also contribute to final admissions “scores.” Such assessments 



10 U.S.C. §§ 4346, 6958, and 9346; and Department of Defense, Directive 1322.22, Service 
Academies, § 4.3, August 24, 1994. 
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include interviews with prospective candidates, teacher/coach 
evaluations, and analyses of writing samples. Though medical criteria 
differ between services, the medical examinations are conducted 
according to the same standards, under a joint DOD Medical Examination 
Review Board that manages the medical examination process and records 
for applicants to all academies. 

Each academy is authorized to permit up to 60 foreign students to attend 
at any given time on a reimbursable basis by their country of origin. This 
number does not count against the authorized student strength of the 
academies. The admission of foreign students is covered by separate 
policies and procedures. Our review was limited to the policies and 
procedures for admitting U.S. citizens to the academies. Figure 1 shows 
the basic steps in the admissions process for all U.S. applicants. 



Figure 1: Basic Steps in the Academy Admissions Process 




See Department of Defense, Directive 5154.25, DOD Medical Examination Review 
Board, June 11, 1981; Directive 6130.3, Physical Standards for Appointment, Enlistment, 
or Induction, December 15, 2000; and Instruction 6130.4, Criteria and Procedure 
Requirements for Physical Standards for Appointment, Enlistment, or Induction in the 
Armed Forces, December 14, 2000. 

10 U.S.C. §§ 4344, 6957, and 9344. 
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United States Military 
Academy Admissions 
Process 



ERIC 



Students who are disenrolled from an academy after the start of their third 
year may be required to complete a period of active duty enhsted service 
of up to 4 years or may be required to reimburse the federal government 
for the cost of their education. Those who are disenrolled in their first 2 
years do not incur an active service or reimbursement obligation.’^ 



The United States Military Academy’s admissions evaluation considers 
academics, leadership, and physical aptitude. Academic considerations 
include above-average high school or college academic records as well as 
strong performance on SAT/ACT. Additionally, the Military Academy 
considers recommendations from Enghsh, mathematics, and science 
teachers. The leadership potential considers demonstrations of leadership 
and initiative in sports, school, community, or church activities and strong 
recommendations from faculty and community leadership and is a more 
subjective assessment of character. Physical aptitude is based on a scored 
standardized test. This test is made up of pull-ups for men or the flexed- 
arm hang for women, push-ups, standing long jump, basketball throw, and 
shuttle run. Figure 2 shows the areas considered and the weights assigned 
to each area in the U.S. Military Academy’s whole person admissions 
process. 



Department of Defense, Directive 1332.23, Service Academy Disenrollment, 
6.2, February 19, 1988. 



( 6.1 and 
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United States Naval 
Academy Admissions 
Process 



Figure 2: Areas and Their Weights Considered in the U.S. Miiitary Academy’s Whole 
Person Admissions Process 




The United States Naval Academy’s admissions evaluation considers 
academics, leadership, physical aptitude, and technical interest. Academic 
considerations include above-average high school or college academic 
records as well as strong performance on SAT/ACT. Additionally, the 
Naval Academy considers recommendations from Enghsh and 
mathematics teachers. Assessment of leadership potential represents a 
subjective evaluation of character in which the academy considers 
demonstrations of leadership in terms of extracurricular activities in 
sports, school, community, or church and strong recommendations from 
faculty and community leadership. Physical aptitude is based on a scored, 
standardized test consisting of pull-ups for men or the flexed-arm hang for 
women, push-ups, standing long jump, basketball throw, and shuttle run. 
Additionally, the Naval Academy considers the technical interest of a 
prospective student, which is measured through a questionnaire in the 
application packet and used to gauge interest in pimsuing a technical 
degree. The intent of this requirement is to admit students that are 
interested in pursuing technical degrees, specifically nuclear and maritime 
engineering. The admissions board can also apply fuitlier points to an 
apphcant’s overall whole person score based on further consideration of 
an apphcant’s record, including such things as the results of the evaluation 
form fiUed out by the Naval Academy representative who interviewed the 
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applicant. Figure 3 shows the areas considered and the weights assigned 
to each area in the U.S. Naval Academy’s whole person admissions 
process. 



Figure 3: Areas and Their Weights Considered in the U.S. Naval Academy’s Whole 
Person Admissions Process 




United States Air Force 
Academy Admissions 
Process 



The United States Air Force Academy’s admissions evaluation considers 
academics, leadership, and an assessment by the selections panel. 
Academic considerations include above-average high school or college 
academic records as well as strong performance on SAT/ACT. 
Additionally, the Air Force Academy considers recommendations from 
English and mathematics teachers. Under leadership potential, the 
academy considers extracurricular activities in sports, school, community, 
or church and strong recommendations from faculty and community 
leadership. Finally, the Air Force Academy Selections Panel makes an 
assessment of all potential students. This assessment is composed of a 
pass/fail score from the physical aptitude examination and the evaluation 
of the academy’s liaison officer evaluation, made after interviewing the 
applicant. The physical aptitude examination is made up of pull-ups for 
men or the flexed-arm hang for women, push-ups, standing long jump, 
basketball throw, and shuttle run. The leadership potential area and the 
admissions board include the more subjective assessments of a potential 
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student. Figure 4 shows the areas considered and the weights assigned to 
each area in the U.S. Air Force Academy's whole person admissions 
process. 



Figure 4: Areas and Their Weights Considered in the U.S. Air Force Academy’s 
Whole Person Admissions Process 



Leadership potential 



Selections panel 



Academic ability 

Source: U.S. Air Force Academy. 




Noniination and The President of the United States alone appoints candidates to the 

of academies.'® Before receiving an appointment, all candidates must secure 

one or more nominations according to the following categories: ” 

• congressional (including a U.S. senator, representative, delegate, or the 
Vice President); 

• service-connected (including, among others, children of disabled 
veterans, erUisted personnel in the active or reserve components, and 
students from ROTC programs or other designated honor school 
graduates); and 



Appointment 

Candidates 



'® 10 U.S.C. §§ 4341a, 6953, and 9341a. 
’’ 10 U.S.C. §§ 4342, 6954, and 9342. 
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• other (including the academy superintendents’ nominees and other 
nominees to bring the incoming class to full strength). 

Figure 5 shows the approximate distribution of categories of academy 
nominations, based on the types and numbers of nominees per category 
allowed by law. 



Figure 5: Categories of Academy Nominations 




Note: Percentages do not add to 100 due to rounding. 



Oversight of the Oversight of the academies is the responsibility of three principal 

Academies organizations: OUSD/P&R, the service headquarters, and the board of 

visitors of each academy. According to Department of Defense Directive 
1322.22 {Service Academies) OUSD/P&R serves as the DOD focal point 
for matters affecting the academies and has responsibility to assess 
academy operations and establish policy and guidance for uniform 
oversight and management of the military academies. The military 
departments perform the primary DOD oversight function for their 
respective academies. The superintendent of each academy reports 



Department of Defense, Directive 1322.22, Service Academies § 5.1, August 24, 1994. 
DOD is currently revising this directive. 
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directly to the uniformed head of his respective service (the Chiefs of Staff 
for the Army and the Air Force and the Chief of Naval Operations for the 
Navy), in accordance with the chain of command for each service. Each 
academy also has a board of visitors, mandated by law,^® that is comprised 
of congressional members and presidential appointees. These boards 
focus attention and action on a wide range of operational and quality of 
life issues at the academies. 

As educational institutions, the service academies are also overseen by 
several nongovernmental organizations that are outside DOD purview. 
Each academy undergoes periodic review by a higher-education 
accreditation body associated with its region of the country, usually 
involving a full review every 10 years with an interim review every 5 years. 
The accreditation bodies review such areas as core curriculum, strategic 
planning, self-assessments, diversity of faculty and students, and faculty 
credentials. The athletic programs of the academies are also subject to 
periodic certification by the National Collegiate Athletic Association. This 
body reviews academy athletics in terms of such issues as finances and 
impact on the education mission of the academies. We limited our review 
of oversight of the academies to DOD organizations and the boards of 
visitors. 



DOD Lacks a 
Complete Framework 
for Oversight of the 
Academies 



The OUSD/P&R, the services, and the academies’ boards of visitors 
conduct many oversight activities, but they lack a complete oversight 
framework. A complete oversight framework includes not only clear roles 
and responsibilities, but also performance goals and measures against 
which to objectively assess performance. Such elements embody the 
principles of effective management in which achievements are tracked in 
comparison with plans, goals, and objectives and the differences between 
actual performance and planned results are analyzed. Without formal 
goals and measures, oversight bodies do not have sufficient focus for their 
efforts and cannot systematically assess an organization’s strengths and 
weaknesses nor identify appropriate remedies that would permit DOD to 
achieve the best value for the investment in the academies. In a prior 



"" 10 U.S.C. §§ 4355, 6968, and 9355. 

Commission on Higher Education of the Middle States Association of Colleges and 
Schools (Military and Naval Academies) and Commission of Institutions of Higher 
Education of the North Central Association of Colleges and Schools (Air Force Academy). 



ERIC 
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report, GAO concluded that better external oversight of the academies 
was needed to provide useful guidance and suggestions for improvement. 
The report recommended that DOD improve oversight of the academies 
through such measures as establishing a focal point for monitoring 
academy issues in the Office of the Secretary of Defense and establishing 
guidance on uniform cost reporting. 

OUSD/P&R and the services have established clear roles and 
responsibilities for oversight of the academies, with the former serving as 
the focal point for issues affecting all academies and the latter having 
direct oversight authority over their respective academies. DOD 
established guidance in 1994 for the oversight of the academies^ and for 
uniform reporting of costs and resources.^ OUSD/P&R is directly involved 
in those policy issues that affect all academies and require DOD-level 
attention and legislative matters. For example, the office was recently the 
DOD focal point on the issue of increasing authorized enrollment at the 
academies from 4,000 to 4,400. With respect to the academies, the office is 
chiefly concerned with monitoring the degree to which the services are 
meeting their goals for the accession of new officers.^^ The office also 
coordinates major studies that affect the academies, such as a November 
1999 report on the career progression of minority and women officers. 

The services are responsible for direct oversight of their respective 
academies; and the academies are treated similarly to major military 
commands. The superintendents of the academies are general/flag officers 
who report directly to the uniformed heads of their services (the Chiefs of 
Staff for the Army and the Air Force and the Chief of Naval Operations for 
the Navy). In addition to overseeing the academies’ budget through the 
same approval process as a major command activity, the services oversee 
the acadenues’ operations and performance primarily through the 
acadenues’ goal of meeting service officer accession targets. The 



U.S. General Accounting Office, DOD Service Academies: Improved Cost and 
Performance Monitoring Needed, GAO/NSIAD-91-79 (Washington, D.C.: July 16, 1991). 

^ Department of Defense, Directive 1322.22, Service Academies, August 24, 1994. DOD is 
currently revising this directive. 

^ Department of Defense, Instruction 1025.4, Service Academy Resources Report, October 
18, 1994. 

^ The academies are one of the sources for officers. The others include reserve officer 
training programs at colleges and universities, officer candidate/training schools, and direct 
commissioning programs. 
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superintendents are responsible for meeting those targets and, in so doing, 
are given wide discretion in such areas as modif3dng their specific 
admissions objectives and the process for matching graduates with service 
assignments. The service headquarters use a munber of mechanisms to 
oversee academy performance. For example, each service headquarters 
provides officer accession targets to the academies so that the assignment 
of graduates and the make up of incoming student classes can be modified 
as necessary. In addition to general munbers of officers, each service also 
has a munber of specialty officer fields that need to be filled, and the 
services also monitor the extent to which the academies will be able to 
meet those accession goals. 

The services also directly oversee the academies by requiring the 
superintendents to report on and discuss their operations. For example, 
the Air Force uses an annual forum of the most senior Air Force officers to 
focus on the Air Force Academy with respect to how it is meeting the 
needs of the operational Air Force. The Navy uses similar senior officer 
conferences and frequent interaction between the superintendent and 
Navy headquEuters to conduct oversight. The Army uses the U.S. Mihtary 
Academy Forum, comprised of senior Army officers, to address academy 
operations issues. The superintendents of the three academies also hold 
annual meetings to discuss issues common to all academies. These 
mechanisms have resulted in such academy actions as curriculiun changes 
to increase the munber of technical degree majors, increasing language 
requirements, and increasing the munber of students attending the 
academies. 

While OUSD/P&R and the services conduct a wide variety of oversight 
activity, there are few stated performance goals against which to measure 
academy operations and performance. Each of the academies has a 
strategic plan that is focused on providing quality mihtary and professional 
training and education in order to commission highly capable junior 
officers. These plans are approved by the service headquarters but are not 
generally used by the services as benchmarks against which to measure 
academy performance, and they do not contain specific goals against 
which to measure student performance. OUSD/P&R is required to assess 
and monitor academy operations based on the information provided in 
annual reports it requires from the service secretaries.^^ These reports 
provide data on various aspects of performance, such as student 



DOD Directive 1322.22 §§ 6.1.2 and 6.1. 
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demographics and trends, student quality, admissions and attrition trends, 
compensation for students and faculty, leadership and honor systems, and 
incidents of indiscipline. 

The reports provide OUSD/P&R and the services with information on 
current and past performance for academy operations, but apart from 
officer accession goals, neither OUSD/P&R nor the services have specific 
stated performance goals against which to compare the information 
provided in the assessment reports, thus they do not have an explicit basis 
forjudging the adequacy of their performance. For exEunple, the data 
collected by the academies show that graduation rates have increased in 
the last 10 years; however, there is no stated goal for a graduation rate 
against which to judge whether this rate of increase is adequate. Other 
data collected by the academies indicate that the percentage of females 
and minorities has fluctuated over the last 3 years, but apart from 
admissions targets used by the U.S. Military Academy, there are no stated 
goals against which to assess these trends. Additionally, academy officials 
regularly analyze data on student body performance to determine the 
extent to which admissions standards can be changed to affect student 
body performance. However, there are no stated goals for student body 
performance, apart from minimum graduation standards, that might help 
the academies and other oversight bodies assess overall student 
performance. 

The oversight efforts of each academy’s board of visitors are similarly 
limited by the absence of sufficient performance goals and measures. Each 
of the academies has a board of visitors, mandated by law^° and comprised 
of Members of Congress and presidential appointees, that is outside the 
DOD chain of command. The boards have a broad legal mandate to inquire 
into all aspects of academy operations.^’ The boards meet several times a 
year to be briefed on and discuss academy operations and must conduct 
an annual visit to their respective academies. During these visits, the 
boards are briefed by academy staff on such issues as admissions, 
curriculum, recruiting, athletics, morale and welfare, and construction 
progrEuns; they also interview students to obtain their perceptions of life at 
the academies. The boards also address inquiries to academy staff, which 
are usually followed up at subsequent meetings, and they make 
suggestions to improve operations or quality of life at the academies. For 



“ 10 U.S.C. §§ 4355, 6968, and 9355. 
10 U.S.C. §§ 4355, 6968, and 9355. 
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example, boards of visitors have recommended increased recruiting of 
qualified minority applicants from various congressional districts and 
increased surveying of students on quality of life issues. 

The boards submit annual reports to the President on the status of and 
issues at the academies but do not evaluate academy operations and 
performance against established performance goals. The boards of visitors 
do not have dedicated staffs to conduct their work, and though board 
members may inquire into any aspect of academy operations, the agenda is 
set largely by the briefings presented to the boards by academy officials. 
Academy officials with whom we spoke were generally satisfied with the 
oversight provided by the boards of visitors, though there were concerns 
at the Air Force Academy about poor attendance by board members 
during annual visits to the academy. 



Whole Person 
Approach Allows 
Academies Flexibihty 
to Admit Students 
with a Range of 
Qualifications 



The academies do not grant waivers from academic criteria but do not 
have absolute minimum scores for admission. Under the whole person 
approach, the academies can admit some applicants whose academic 
scores are lower than might normally be competitive for admission, but 
who in their totality (academics, physical aptitude, and leadership 
potential) are deemed an acceptable risk and qualified to attend an 
academy. This admissions approach is consistent with the intent of the 
academies to admit students who also demonstrate leadership and 
initiative characteristics, which cannot be quantified by purely objective 
scoring methods. 



When conducting their admissions processes, the academies do not set 
absolute minimum scores for academic ability. Rather, they establish a 
range of scores that would be considered competitive, based on past 
incoming class performance and academy research on the overall quality 
of the applicant pool. Prior to 2002, the Air Force Academy set absolute 
minimum academic scores, and a waiver was required to further consider 
an applicant who fell below that minimum, no matter how high his or her 
scores in the leadership area. However, the Air Force Academy no longer 
has absolute minimums and uses the same competitive range approach as 
the other academies. Under this approach, if an applicant’s academic score 
is lower than the competitive range guidelines, academy officials have 
some flexibility to further consider the applicant. Academy officials will 
re-examine the applicant’s record for information that might provide 
further insight about his or her academic achievement. For example, 
officials may contact high school teachers to inquire about the types and 
difficulty of the classes the applicant has been taking and his or her 
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performance in those classes. Academy officials will also weigh the extent 
to which the leadership component of the applicant’s whole person score 
offset the low component. The applicant is considered a risk and is 
evaluated through a deUberative process by academy officials on the basis 
of their judgment of whether the appUcant is fully qualified and capable of 
succeeding at that academy. The subjective natmre of this approach is 
consistent with the intent of the whole person concept, by which the 
academies want to admit students who also demonstrate leadership 
characteristics that cannot be quantified by purely objective scoring 
methods. Academy officials do not consider these judgments to constitute 
a waiver of academic standards, but rather a judicious assessment of the 
whole person. The process for assessing those applicants whose academic 
scores are lower than might normally be competitive is nonetheless similar 
to the former Air Force Academy process for granting waivers. 

With over 10,000 applicants^® for each academy each year and about 1,200 
students admitted, the academic standards are high. Academy data show 
that the academic quality of the appUcants has remained high over the past 
4 years, and the competitive ranges for academic scores used by the 
academies have remained the same or have increased during this time. 
However, it is possible for students to be admitted whose academic scores 
were not as competitive as some other applicants who may not have been 
admitted. Senators, representatives, and delegates may submit up to 10 
nominees for each student vacancy available to him or her per academy. 
They may choose to designate one as a principal nominee.^® If an appUcant 
receives a principal nomination and is in all other respects qualified, the 
academies must admit that appUcant, even over an appUcant on the same 
senator’s, delegate’s, or representative’s nomination Ust with higher 
academic and/or whole person scores. The other nominated names 
become alternates for possible admission later in the admissions process. 

Though some academies award credit for the extent to which an appUcant 
siupasses the standards of the physical aptitude examination, there are 
minimum standards for the physical test that must be met. None of the 
academies uses a system of “waivers,” except for medical conditions. An 
appUcant can be waived for a medical condition, based on the deUberation 
and judgment of DOD medical personnel and the academy superintendent. 



^ This includes the total number of students who applied and not the number that received 
a nomination. 

“ 10 U.S.C. §§ 4342, 6954, and 9342. 
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For example, an applicant who is disqualified due to a vision condition 
may apply for and receive a waiver, based on subsequent surgical vision 
correction or determination by the academy superintendent that the 
applicant would be able to serve on active duty without the vision 
condition being a problem. 



No Significant 
Differences in 
Admissions and 
Academy 

Performance between 
Various Groups of 
Students 



In our review of the academy classes that started in 1998 (class of 2002), 
we found differences among various groups of students in their 
admissions scores and similar differences in their performance while at 
the academies, but the differences were not significant in magnitude. In 
terms of performance after admission to the academies,^^ differences 
between these student groups and the class as a whole were also not 
sizable. We reviewed data for the following distinct groups: 

• overall class, 

• females, 

• minorities, 

• academy preparatory school graduates, 

• recruited athletes, 

• prior enlisted, and 

• lower 30 percent of class by academic admissions scores. 



For the class data we reviewed, minorities, academy preparatory school 
graduates, recruited athletes, and prior enlisted students all had lower 
average admissions scores than the average for the class as a whole, 
though these differences varied. The differences between groups and the 
class as a whole were not sizable, generally falling within 5 percent. Those 
differences that were statistically significant and outside the 5 percent 
range were still generally less than 10 percent of the class as a whole. 
Tables 3, 4, and 5, show the average admissions scores for the selected 
groups in the class that started in 1998 at the Military, Naval, and Air Force 
Academies, respectively. Although each academy uses the same 
fundamental whole person approach, they use different scales to calculate 
scores. Therefore, the academic and whole person scores cannot be 
compared across academies. 



We used the following performance factors to measure student performance at the 
academies: cumulative grade point average, cumulative military performance average, 
order of merit (class rank), and graduation rate (for each group of students). 

Each of these groups can contain members from other groups. 
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Table 3: Average Admissions Scores for the Seiected Groups in the Ciass That Started in 1998 at the U.S. Miiitary Academy 



Lower 

Preparatory 30 percent of 

schooi Recruited admissions 



Average 

admissions score 


Overall 

(1,246) 


Females 

(192) 


Minorities 

(269) 


graduates 

(184) 


athletes 

(279) 


Prior enlisted 
(31) 


class 

(377) 


Acadennic score 


600 


603 


583 


546" 


558“ 


594 


532" 


Whole person score 


6,006 


6,022 


5,865 


5,645“ 


5,814 


5,861 


5,609“ 



Source: GAO analysis, from Military Academy sources. 

"Denotes a group average or percentage with a statistically significant difference greater than 5% 
from the overall average or percentage. 

“’Denotes a group average or percentage with a statistically significant difference greater than 10% 
from the overall average or percentage. 



Table 4: Average Admissions Scores for the Seiected Groups in the Ciass That Started in 1998 at the U.S. Navai Academy 



Lower 

Preparatory 30 percent of 

school Recruited admissions 



Average 

admissions score 


Overall 

(1,226) 


Females 

(190) 


Minorities 

(221) 


graduates 

(146) 


athletes 

(380) 


Prior enlisted 
(76) 


class 

(368) 


Acadennic score 


618 


624 


594 


545" 


596 


570“ 


544" 


Whole person Score 


65,732 


65,719 


63,769 


61,254“ 


64,233 


62,256“ 


61,404“ 



Source: GAO analysis, from Naval Academy sources. 

‘Denotes a group average or percentage with a statistically significant difference greater than 5% 
from the overall average or percentage. 

“’Denotes a group average or percentage with a statistically significant difference greater than 10% 
from the overall average or percentage. 



Table 5: Average Admissions Scores for the Selected Groups in the Class That Started in 1998 at the U.S. Air Force Academy 



Lower 

Preparatory 30 percent of 

school Recruited admissions 



Average 

admissions score 


Overall 

(1,216) 


Females 

(190) 


Minorities 

(229) 


graduates 

(157) 


athletes 

(312) 


Prior enlisted 
(44) 


class 

(366) 


Academic Score 


3,202 


3,216 


3,123 


3,112 


3,043 


3,188 


2,863" 


Whole person Score 


798 


805 


782 


774 


773 


792 


751“ 



Source: GAO analysis, from Air Force Academy sources. 

‘Denotes a group average or percentage with a statistically significant difference greater than 5% 
from the overall average or percentage. 

“’Denotes a group average or percentage with a statistically significant difference greater than 10% 
from the overall average or percentage. 
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Of those students in the lower 30 percent of the class in terms of academic 
admissions scores, about 44 percent were recruited athletes, between 25 
and 31 percent were minorities, and between 20 and 34 percent were 
preparatory school graduates. Table 6 shows the percentage of the 
selected groups making up the lower 30 percent of the classes in terms of 
their academic admissions scores, by academy. 



Table 6: Percentage of the Selected Groups Making Up the Lower 30 percent of the Classes in Terms of Their Academic 
Admissions Scores, by Academy 



Numbers in percent 


Academy 


Females 


Minorities 


Preparatory 
school graduates 


Recruited 

athletes 


Prior enlisted 


Military Academy 


16 


31 


34 


44 


3 


Naval Academy 


13 


29 


32 


45 


13 


Air Force Academy 


14 


25 


20 


44 


5 


Source: GAO analysis, from DOD sources. 



Note: Numbers do not add to 100 percent because each of the groups can contain members of 
another group. 



We also found differences in performance after admission to the 
academies between selected groups and the class as a whole. For 
example, females at the Naval Academy had a lower graduation rate than 
the class as a whole, but they had a higher average academic grade point 
average (cumulative GPA) than the class as a whole and higher average 
class rank (order of merit). The differences in performance between the 
selected groups and the class as a whole were not sizable, generally falling 
within 5 percent. Those differences that were statistically significant and 
outside the 5 percent range were still generally less than 10 percent of the 
class as a whole. Tables 7, 8, and 9 show how the selected groups 
performed at the Military, Naval, and Air Force Academies, respectively. 
See appendix 11 for fiuther information on comparisons of performance by 
defined student groups. 
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Table 7: Student Performance for Selected Groups at the U.S. Military Academy 



Performance score 


Overall 

(1,246) 


Females 

(192) 


Minorities 

(269) 


Preparatory 

school 

graduates 

(184) 


Recruited 

athletes 

(279) 


Prior enlisted 
(31) 


Lower 
30 percent of 
admissions 
class 
(377) 


Average cumulative 
GPA 


2.99 


2.99 


2.82 


2.61" 


2.81 


3.14 


2.66" 


Average cumulative 
MPA 


3.28 


3.26 


3.21 


3.26 


3.20 


3.37 


3.21 


Average order of merit 


3.03 


3.04 


2.86" 


2.75" 


2.90 


3.06 


2.78" 


Graduation rate 


78% 


76% 


71 %" 


72% 


76% 


71% 


71 %" 


Source: GAO analysis, from Military Academy sources. 



‘uenoies a group average or percenlage with a statistlcaHy significant difference greater than 5% 
from the overall average or percentage. 



“’Denotes a group average or percentage with a statistically significant difference greater than 10% 
from the overall average or percentage. 



Table 8: Student Performance for Selected Groups at the U.S. Naval Academy 



Performance score 


Overall 

(1,226) 


Females 

(190) 


Minorities 

(221) 


Preparatory 

school 

graduates 

(146) 


Recruited 

athletes 

(380) 


Prior enlisted 
(76) 


Lower 
30 percent of 
admissions 
class 
(368) 


Average cumulative 
GPA 


2.97 


3.01 


2.82" 


2.67" 


2.86 


3.02 


2.67" 


Average cumulative 
MPA 


3.12 


3.16 


3.02 


2.99 


3.08 


3.19 


3.00 


Average order of merit 


489 


456" 


590*’ 


658“ 


551 


453 


661“ 


Graduation rate 


80% 


71%“ 


75% 


77% 


79% 


72% 


76%" 


Source: GAO analysis, from Naval Academy sources. 



“Denotes a group average or percentage with a statistically significant difference greater than 5% 
from the overall average or percentage. 



““Denotes a group average or percentage with a statistically significant difference greater than 10% 
from the overall average or percentage. 
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Table 9: Student Performance for Selected Groups at the U.S. Air Force Academy 



Performance score 


Overall 

(1,216) 


Females 

(190) 


Minorities 

(229) 


Preparatory 

school 

graduates 

(157) 


Recruited 

athletes 

(312) 


Prior enlisted 
(44) 


Lower 
30 percent of 
admissions 
class 
(366) 


Average cumulative 
GPA 


2.93 


2.97 


2.78“ 


2.61" 


2.79 


2.89 


2.64“ 


Average cumulative 
MPA 


2.90 


2.93 


2.89 


2.83 


2.81 


2.93 


2.84 


Average order of merit 


469 


440 


545" 


663" 


568" 


499 


646" 


Graduation rate 


74% 


75% 


71% 


69% 


71% 


66% 


65%" 



Source: GAO analysis, from Air Force Academy sources. 



“Denotes a group average or percentage with a stalisticaiiy significant difference greater than 5% 
from the overall average or percentage. 

"Denotes a group average or percentage with a statistically significant difference greater than 10% 
from the overall average or percentage. 

Some groups — such as minorities, preparatory school graduates, recruited 
athletes, and students in the lower 30 percent of their class in terms of 
academic admissions scores — ^performed at lower levels on average in all 
categories than the class as a whole, but these differences varied between 
academies and by category and were not sizable. For example, one of the 
lowest average academic grade point averages for the groups we reviewed 
was 2.61 and the average for the class as a whole at that academy was 2.93. 
A 2.0 grade point average is required to graduate for academic and military 
averages. Similarly, the lowest graduation rate for the class we reviewed 
was 65 percent for the students in the lower 30 percent of their class in 
terms of academic admissions scores at one academy. The average 
graduation rate for the class as a whole was 74 percent. 

Our analysis of data for the students who entered the academies in 1998 
(class of 2002) indicates that admissions scores are generally good 
predictors of performance at the academies. Of the admissions scores, the 
academic component of the whole person scores was often the best 
predictor of academic performance at the academies, and the whole 
person scores in their entirety were often the best predictors of mUitaiy 
performance at the academies. Both academic and whole person 
admissions scores were good predictors of class rank. In general, whole 
person admissions scores were better predictors of graduation rate than 
the academic admissions scores alone. 
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Conclusion 



Although the service academies receive oversight from a number of 
organizations and have estabhshed guidance for that oversight that 
includes the reporting of a vdde range of data on academy operations, 
without clear and agreed-upon performance goals, there is no objective 
yardstick against which to fully measure academy performance and 
operations, apart from the officer accessions goals currently used. 
Establishment of such performance goals is consistent with the principles 
of effective management and would enhance the quality of oversight 
already performed by OUSD/P&R, the services, and the academy boards of 
visitors, permitting them to more clearly note those areas in which the 
academies excel, highhght areas where improvement is warranted, and 
achieve the best value for the nation’s investment in the academies. 



Recommendation for 
Executive Action 



To improve DOD oversight of the operations and performance of the 
service academies, we recommend that the Secretary of Defense direct the 
OUSD/P&R, in concert vdth the services, to fiuther enhance performance 
goals and measures whereby the information required in annual 
assessment reports can be better evaluated. These performance goals 
should be developed for each academy and, where appropriate, in 
common for all academies. The specific goals should coincide with 
performance elements agreed upon by the services and OUSD/P&R and 
might include such things as graduation rates, demographic composition 
of student classes, assessments of officer performance after graduation, 
and other performance information already collected by the academies, 
including performance characteristics of various groups of students. 



Agency Comments 



In comments on a draft of this report, DOD agreed with our 
recommendation to further enhance performance goals and measures for 
the service academies whereby the information required in annual 
assessment reports can be better evaluated. DOD fmther stated that the 
Office of the Under Secretary of Defense for Personnel and Readiness 
OUSD/P&R will (1) monitor development of improved goals and measures 
by the service academies, to include facUitating the development of 
common performance goals where appropriate and (2) update DOD 
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Directive 1322.22, Service Academies, as required. DOD’s written 
comments are included in their entirety in appendix III. 



We are sending copies of this report to the appropriate congressional 
committees; the Secretaries of Defense, the Army, the Navy, and the Air 
Force; and the Director, Office of Management and Budget. We will also 
make copies available to others upon request. In addition, the report will 
be available at no charge on the GAO Web site at http://www.gao.gov. 

Please contact me on (202) 512-5559 if you or your staff have any 
questions concerning this report. Key contributors are hsted in 
appendix V. 




Derek B. Stewart 
Director 

Defense Capabilities and Management 
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Appendix I: Scope and Methodology 



To assess the extent to which DOD oversees the service academies’ 
operations and performance, we interviewed officials at the Office of the 
Under Secretciry of Defense for Personnel and Readiness; the Army, Navy, 
and Air Force headquarters; and the U.S. Military, U.S. Naval, and U.S. Air 
Force Academies. We reviewed documents on service and DOD oversight 
criteria and structmres, reporting mechanisms, academy strategic plans, 
academy annual reports on operations and performance, boards of 
visitors’ minutes and reports, and superintendents conference reports. We 
also attended a U.S. Naval Academy Board of Visitors meeting at the Naval 
Academy in December 2002 and a U.S. Military Academy Board of Visitors 
meeting in Washington, D.C., in March 2003. Additionally, we reviewed 
criteria on the principles of effective management, such as those found in 
Standards for Internal Control in the Federal Government! 

To assess the extent to which academy applicants are granted waivers 
from academic admissions criteria, we interviewed officials from the 
Military, Naval, and Air Force Academies and reviewed documents on 
admissions policies, standards, and practices. We discussed with academy 
officials their execution of the whole person approach, including how they 
assess applicants’ records, the weights applied to the various components 
of the whole person score (academic, leadership, and physical aptitude), 
and the justification for points given to various aspects of an applicant’s 
scores. We also reviewed data from each academy on trends in academic 
admissions scores. During site visits to each academy, we observed the 
evaluation of applicant packages for the incoming class of 2007 by 
academy officials, including how the whole person approach was applied 
for admissions scores. We also observed meetings of senior officials at 
each academy where applicants’ records were evaluated and final 
admissions decisions were made. 

To assess the extent to which admissions and academy performance 
scores differ between various groups of students, we analyzed admissions 
scores and academy performance scores for all students who started at 
the three academies in 1998 and should have graduated in 2002. This 
represented the most recent group of students for which complete data 
were available. We requested and received from each academy a database 
that included data on both admission scores and information about 
students’ performance while attending the academy. We did not 



' U.S. General Accounting Office, Standards for Internal Control in the Federal 
Government, GAO/AIMD-00-21.3.1 (Washington, D.C.; November 1999). 



o 
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Appendix I: Scope and Methodology 



independently assess data reliability, but we obtained assurances about 
data completeness, accuracy, and reliability from academy officials 
responsible for maintaining data at each academy. We analyzed these data 
separately for each academy since each academy calculated admission 
scores or performance scores somewhat differently. We identified six 
major groups of students conunon to all academies: females, minorities, 
academy preparatory school graduates, recruited athletes, prior enlisted 
personnel, and students whose academic admission scores fell in the 
lower 30 percent of the entering class (we chose the latter group in order 
to capture information on students whose academic admissions scores 
may have been lower than might normally be competitive). Information 
specifying a student’s membership in each of these groups was provided in 
the databases from the academies. To assess differences, we first 
compared the mean performance scores for each group to the overall 
mean for each performance measure for the entire class. See appendix II 
for details on the results of our analysis of the relationships between 
admissions and performance scores. 

In addition, we assessed the relationship between admissions scores and 
performance at the academies by using the whole person admission score 
and the academic component of the admissions score. We estimated the 
effects of those scores on four measures of performance for students at 
the academies: (1) cumulative grade point average (GPA), (2) cumulative 
military performance average (MPA), (3) order of merit (class standing), 
and (4) graduation rate. We used cumulative GPA upon graduation as an 
indicator of academic performance at the academies and military 
performance averages upon graduation as an indicator of military 
performance at the academies. Order of merit is a measure of class 
standing at each academy that combines academic and military grade 
performance and is a final rank for each graduating student. At both the 
Air Force Academy and the Naval Academy, order of merit is an actual 
class rank number. At the Military Academy, however, order of merit 
could range between 0 and 4.0 and was given on the same scale as grade 
point averages. For each academy, we analyzed the association of both the 
academic component scores and whole person admission scores with 
each of the performance scores using regression models. Relationships 
between the admissions scores and cumulative GPA, cumulative MPA, and 
order of merit were estimated using linear regression models. The 
relationships between these two admissions scores and the likelihood of 
graduating were estimated using logistic regression models. See appendix 
II for more details on the results of those analyses. 
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Appendix I: Scope and Methodology 



Issues related to alleged sexual assaults at the academies fell outside the 
scope of om objectives. We conducted om work from October 2002 
through May 2003 in accordance with generally accepted government 
auditing standards. 
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Appendix II: Results of Statistical Analysis of 
Class of 2002 Admissions and Academy 
Performance Scores 



This appendix provides the results of our analyses of both admissions and 
performance scores for the class of 2002 at the U.S. Military Academy, the 
U.S. Naval Academy, and the U.S. Air Force Academy. 



Admissions and 
Performance Scores 



We obtained data from all three service academies that included 
information on admissions scores (academic and whole person), 
performance scores while at the academy (cumulative academic grade 
point average, military performance average, and order of merit), attrition 
information where applicable, and various demographic characteristics for 
all students entering each academy in 1998, Table 10 shows the rniriimum, 
maximum and average admissions and performance scores for students at 
each academy. Table 11 shows graduation rates at each academy. 



Table 10: Admissions and Academy Performance Scores for the Class of 2002 



Military Academy Naval Academy Air Force Academy 





Average 


Min 


Max 


Average 


Min 


Max 


Average 


Min 


Max 


Academic 
Admissions score 


600 


430 


791 


618 


440 


788 


3,202 


2,492 


4,005 


Whole person score 


6,006 


4,587 


7,188 


65,732 


51,651 


82,250 


798 


655 


931 


Cum. GPA 


2.99 


1.97 


4.19 


2.97 


2.03 


4.00 


2.93 


2.06 


3.97 


Cum. MPA 


3.28 


2.09 


3.99 


3.12 


2.17 


3.85 


2.90 


2.32 


3.92 


Order of merit 


3.03 


1.30 


3.92 


489 


1 


977 


469 


1 


929 



Source; GAO analysis, from DOD sources. 

Note: For the U.S. Air Force Academy, an additional step during the selection panel process results In 
a lower whole person score than the component parts. 



Table 11 : Number of Students Graduating and Graduation Rates for the Class of 2002 





Military Academy 


Naval Academy 




Air Force Academy 




Number 


Percent 


Number 


Percent 


Number Percent 


Graduation rate 


968 


78% 


977 


80% 


894 74% 


Source: GAO analysis, from dOD sources. 



Next, we compared the average admissions scores, performance scores, 
and graduation rates of the six student groups to these overall scores and 
rates. Tables 12, 13, and 14 show the average admission scores and the 
four measures of student performance for the overall sample, and for the 
six student groups, for each of the academies. Because we have data for 
the population of students in this class and there is no sampling error, the 
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Appendix II: Results of Statistical Analysis of 
Class of 2002 Admissions and Academy 
Performance Scores 



standard error of these estimates are small and differences that could be 
considered small in magnitude may in fact be statistically significant. In 
the tables below, differences that are statistically significant (p<.05) and 
exceed 5 percent are considered meaningful and noted, though such 
differences may not be practically significant when compared with class 
performance requirements overall. For example, at the Naval Academy the 
overall average academic admissions score is 618, 5 percent of 618 is 
about 31. Only those group average academic admissions scores that are 
statistically significant and more than 31 points below 618 are noted with 
an “a.” Differences that are greater than 10 percent are marked with a “b.** 



Table 12: Admissions and Performance Scores for the Class of 2002 at the U.S. Military Academy 



Lower 30 
percent of 

Prep school Recruited Prior eniisted admissions 





Overall 

(1,246) 


Females 

(192) 


Minorities 

(269) 


graduates 

(184) 


athletes 

(279) 


personnel 

(31) 


class 

(377) 


Academic admissions 


600 


603 


583 


546" 


558“ 


594 


532" 


Whole person admissions score 


6,006 


6,022 


5,865 


5,645“ 


5,814 


5,861 


5,609“ 


Four performance measures 


1. Cumulative GPA 


2.99 


2.99 


2.82 


2.61“ 


2.81 


3.14 


2.66“ 


2. Cumulative MPA 


3.28 


3.26 


3.21 


3.26 


3.20 


3.37 


3.21 


3. Order of merit 


3.03 


3.04 


2.86“ 


2.75“ 


2.90 


3.06 


2.78“ 


4. Graduation rate 


78% 


76% 


71%“ 


72% 


76% 


71% 


71%“ 


Source; GAO analysis, from Military Academy sources. 



'Denotes a group average or percentage with a statistically significant (p<.05) difference greater than 
5% from the overall average or percentage. 

‘'Denotes a group average or percentage with a statistically significant (p<.05) difference greater than 
10% from the overall average or percentage. 



o 
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Appendix 11: Results of Statistical Analysis of 
Class of 2002 Admissions and Academy 
Performance Scores 



Table 13: Admissions and Performance Scores for the Class of 2002 at the U.S. Naval Academy 



Lower 30 
percent of 

Prep school Recruited Prior enlisted admissions 





Overall 

(1,226) 


Females 

(190) 


Minorities 

(221) 


graduates 

(146) 


athletes 

(380) 


personnel 

(76) 


class 

(368) 


Academic admissions 


618 


624 


594 


545“ 


596 


570“ 


544'’ 


Whole Person admissions score 


65,732 


65,719 


63,769 


61,254“ 


64,233 


62,256“ 


61,404“ 


Four performance measures 


1. Cumulative GPA 


2.97 


3.01 


2.82“ 


2.67“ 


2.86 


3.02 


2.67“ 


2. Cumulative MPA 


3.12 


3.16 


3.02 


2.99 


3.08 


3.19 


3.00 


3. Order of merit 


489 


456“ 


590'^ 


658“ 


55f 


453 


661 


4. Graduation rate 


80% 


71%“ 


75% 


77% 


79% 


72% 


76%“ 


Source: GAO analysis, from Naval Academy sources. 



‘Denotes a group average or percentage with a statistically significant (p<.05) difference greater than 
5% from the overall average or percentage. 

'’Denotes a group average or percentage with a statistically significant (p<.05) difference greater than 
10% from the overall average or percentage. 



Table 14: Admissions and Performance Scores for the Class of 2002 at the U.S. Air Force Academy 



Lower 30 
percent of 

Prep school Recruited Prior enlisted admissions 





Overall 

(1,216) 


Females 

(190) 


Minorities 

(229) 


graduates 

(157) 


athletes 

(312) 


personnel 

(44) 


class 

(366) 


Academic admissions 


3,202 


3,216 


3,123 


3,112 


3,043 


3,188 


2,863“ 


Whole Person admissions score 


798 


805 


782 


774 


773 


792 


751“ 


Four performance measures 


1. Cumulative GPA 


2.93 


2.97 


2.78“ 


2.61“ 


2.79 


2.89 


2.64“ 


2. Cumulative MPA 


2.90 


2.93 


2.89 


2.83 


2.81 


2.93 


2.84 


3. Order of merit 


469 


440 


545“ 


663“ 


568“ 


499 


646“ 


4. Graduation rate 


74% 


75% 


71% 


69% 


71% 


66% 


65%“ 


Source: GAO analysis, from Air Force Academy sources. 



‘Denotes a group average or percentage with a statistically significant (p<.05) difference greater than 
5% from the overall average or percentage. 

'’Denotes a group average or percentage with a statistically significant (p<.05) difference greater than 
10% from the overall average or percentage. 
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Appendix 11: Results of Statistical Analysis of 
Class of 2002 Admissions and Academy 
Performance Scores 



Relationships 
between Admissions 
and Performance 
Scores 



ERIC 



Regression models were used to assess the relationship between 
admission scores and performance at the three academies. We used linear 
regression models to examine relationships between admission scores and 
GPA, MPA, and order of merit. To examine the relationship between 
admission scores and the Ukelihood of graduating we used a logistic 
regression model. Both the academic admission score and the whole 
person score were included as independent veuiables in each model. We 
estimated separate regression models for each academy. The results of 
these regressions are shown in tables 15 and 16. 

The tables show both regression coefficients and standardized 
coefficients. In general, regression coefficients are interpreted as the 
predicted change in the dependent veuiable for every unit change in the 
independent veuiables. Here, we have scaled the admissions scores so that 
the regression coefficients in the table can be interpreted as the predicted 
change in the relevant measime of success for every 100-point increase in 
the academic or “whole person” admission score. For example, overall at 
the U.S. Air Force Academy, for every 100-point increase in the academic 
admission score we expect to see a 0.06 increase in GPA. For every 100- 
point increase in the “whole person” score, we expect to see a 0.18 
increase in GPA. Both relationships are statistically significant, meaning 
that both the academic score and the “whole person” score are significant 
predictors of cumulative GPA at the academy. 

We cannot compare the size of these coefficients across the three 
academies, though, because the academic and “whole person” scores are 
on different scales. Because the size of the unstandardized regression 
coefficients is affected by the scale of the independent veuiables (the 
admissions scores), we use standardized regression coefficients to 
compare them. These appear in parentheses in the tables. To estimate 
these coefficients, all of the coefficients are standardized by dividing the 
regression coefficient by the ratio of the standard deviation of the success 
measime to standard deviation of the admission score. The standardized 
regression coefficients, therefore, represent the change in the measure of 
success for each change of one standard deviation in admission scores. 
Using standardized coefficients, one can conclude that the coefficient that 
is larger in magnitude has a greater effect on the measures of success. 
Using the same U.S. Air Force Academy example, we see that while the 
relationships between both academic and “whole person” scores and GPA 
are significant, the relationship between academic scores and GPA is 
actually a stronger one than the relationship between the “whole person” 
score and GPA. Overall, while the academic scores are often a better 
predictor of academic performance at the academies (GPA), the “whole 
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Appendix II: Results of Statistical Analysis of 
Class of 2002 Admissions and Academy 
Performance Scores 



person” scores are often better predictors of military performance (MPA). 
The academic admissions scores have no effect on MPA at the Mihtary and 
Air Force Academies and the whole person scores, not the academic 
admissions scores, predict likelihood of graduating at all three academies. 

We also used the statistic to estimate how much of the variation in each 

performance score can be explained by both academic and whole person 
admission scores. The admission scores explained about 30 percent of the 
variation in GPAs at both the Naval and Air Force Academies and about 40 
percent of the variation in GPAs at the Military Academy. The admission 
scores explained between a quarter and a third of the variation in order of 
merit across the three academies. However, admission scores did not 
explain as much of the variation in either mihtary performance scores or 
graduation rates. Therefore, while both types of admission scores are 
significant predictors of performance at the academy, they only explain 
between 7 and 40 percent of the variation in performance at the 
academies, and only a very smah percentage of the variability in the 
Ukelihood of graduating. Other factors not studied here, such as the 
mihtary training and academic environment students experience at the 
academies, may contribute to performance more than just students’ 
admissions scores do. 



Table 15: Regression Coefficients (Standardized Coefficients) from Linear Regression Models Testing Correlations between 
Academic and Whole Person Admissions Scores with Cumulative GPA, Cumulative MPA, and Order of Merit for the Class of 
2002 at the Service Academies 




Cumulative GPA 


Cumulative MPA 




Order of Merit 




Academic Whole person 

admission score score 


Academic Whole person 

admission score score 


Academic Whole person score 

admission score 


Military Academy 


.42 


.01 


-.04 


.03 


.18 




.02 


class overall 


(.56)- 


(.09) 


(-.11) 


(.44)- 


(.34)“ 




(.29)" 




= .42 




R^ = .12 






R^ = .37 




Naval Academy 


.28 


.002 


.09 


.001 


-157.62 




-1.13 


class overall 


(.38)" 


(.18) 


(.18)- 


(.19)- 


(-.35)“ 




(-.17)“ 




= .30 




R^=.13 






CO 

C\J 

II 

a: 




Air Force Academy 


.06 


.18 


-.01 


.21 


-27.89 




-147.29 


class overall 


(.38)" 


(.20)- 


(-.14) 


(.37)“ 


(-.30)" 




(-.25)“ 




R^ = .31 




R^ = .07 






R" = .29 





Source: GAO analysis, from DOD sources. 



Note: Because of the difference in scales for admissions scores between academies, the size of the 
coefficients cannot be compared across academies. 

“ Denotes statistically significant (p<-05) relationships. 
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Appendix II: Results of Statistical Analysis of 
Class of 2002 Admissions and Academy 
Performance Scores 



Table 16: Regression Coefficients (Standardized Coefficients) from Logistic 
Regression Models Testing Correlations Between Academic and Whole Person 
Admissions Scores and the Likelihood of Graduation for the Class of 2002 at the 
Service Academies 



Graduation 



Academic admission score Whole person score 


Military Academy class 


-.11 




.11 


overall 


(-.03) 




(.24)“ 






R" = .02 




Naval Academy class 


-.36 




.01 


overall 


(-.13) 




(.23)“ 






R" = .01 




Air Force Academy class 


.01 




.75 


overall 


(.02) 




(.20)“ 



R" = .03 



Source: GAO analysis, from DOD sources. 

Note: Because of the difference in scales for admissions scores between 
academies, the size of the coefficients cannot be compared across academies. 

“ Denotes statistically significant (p<.05) relationships. 
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Appendix III: Comments from the 
Department of Defense 




OFFICE OF THE UNDER SECRETARY OF DEFENSE 
4CXX> DEFENSE PENTAGON 
WASHINGTON, D.C. 20301-4000 



AUG 2 5 2003 




PCnSONNEL AND 
RCAOlNESa 



Mr. Derek B. Stewart 

Director, Defense Capabilities and Management 
United States General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Stewart: 

This is the Department of Defense (DoD) response to the General Accounting 
Office (GAO) draft report, GAO-03-1000, “MILITARY EDUCATION: DoD Needs to 
Enhance Performance Goals and Measures to Improve Oversight of Military Academies,’* 
dated July 31, 2003 (GAO Code 350270). 

The Department concurs with the report’s conclusion that the Office of the Under 
Secretary of Defense for Personnel and Readiness, in concert with the Services, should 
further enhance performance goals and measures for the Service academies whereby the 
information required in annual assessment reports can be better evaluated. 

Numerous current guidelines regulate the performance of the Service academies. 
However, the lack of specific performance goals in precise areas such as graduation or 
attrition rates prevents full analysis of the annual data collected. Therefore, performance 
goals should be developed for each academy and, where appropriate, in common for all 
academies to address these, and potentially other, areas of interest. 

The enclosure addresses the specific recommendation made by the GAO. The 
Department appreciates the opportunity to comment on the draft report. 



Sincerely, 




tTharles S. Abell 
Principal Deputy 



Enclosure: 
As stated 
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Appendix III: Comments from the Department 
of Defense 



GAO-03-1000/GAO CODE 350270 

“MILITARY EDUCATION: DOD NEEDS TO ENHANCE PERFORMANCE GOALS 
AND MEASURES TO IMPROVE OVERSIGHT OF 
MILITARY ACADEMIES” 

DEPARTMENT OF DEFENSE COMMENTS 
TO THE RECOMMENDATION 



RECOMMENDATION : The GAO recommended that the Secretary of Defense direct 
the Office of the Under Secretary of Defense for Personnel and Readiness (OUSD/P&R), 
in concert with the Services, to further enhance performance goals and measures whereby 
the information required in annual assessment reports can be better evaluated. These 
performance goals should be developed for each academy and, where appropriate, in 
common for all academies. The specific goals should coincide with performance 
elements agreed upon by the Services and OUSD/P&R and might include such things as 
graduation rates, demographic composition of student classes, assessments of officer 
performance after graduation, and other performance information already collected by the 
academies, including performance characteristics of various groups of students. (Page 
32/Draft Report). 

DoD RESPONSE: Concur. The development of better measures by OUSD(P&R) in 
conjunction with the Military Departments to assist in assessment of the Service 
academies will enhance performance goals and allow more useful evaluation of the 
information currently required in the annual assessment reports. OUSD(P&R) will 
monitor development of improved goals and measures by the Service academies, to 
include facilitating the development of common performance goals where appropriate, 
and will update Department of Defense Directive 1322.22, Service Academies, as 
required. ECD is March 2004. 
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GAO’s Mission 


The General Accounting Office, the audit, evaluation and investigative arm of 
Congress, exists to support Congress in meeting its constitutional responsibilities 
and to help improve the performance and accountability of the federal 
government for the American people. GAO examines the use of public funds; 
evaluates federal programs and policies; and provides analyses, 
recommendations, and other assistance to help Congress make informed 
oversight, policy, and funding decisions. GAO’s commitment to good government 
is reflected in its core values of accountability, integrity, and reliability. 


Obtaining Copies of 
GAO Reports and 
Testimony 


The fastest and easiest way to obtain copies of GAO documents at no cost is 
through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- 
text files of current reports and testimony and an expanding archive of older 
products. The Web site features a search engine to help you locate documents 
using key words and phrases. You can print these documents in their entirety, 
including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 
daily. The list contains links to the full-text document files. To have GAO e-mail 
this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail 
alerts” under the “Order GAO Products” heading. 


Order by Mail or Phone 


The first copy of each printed report is free. Additional copies are $2 each. A 
check or money order should be made out to the Superintendent of Documents. 
GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a 
single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 
441 G Street NW, Room LM 
Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 


To Report Fraud, 
Waste, and Abuse in 
Federal Programs 


Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 
E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470 


Public Affairs 


Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 
U.S. General Accounting Office, 441 G Street NW, Room 7149 
Washington, D.C. 20548 
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